Lessons Learned from the CHiC and SBS Interactive Tracks: A Wishlist for Interactive IR Evaluation

Over the course of the past two decades, the Interactive Tracks at TREC and INEX have contributed greatly to our knowledge of how to run an interactive IR evaluation campaign. In this position paper, we add to this body of knowledge by taking stock of our own experiences and challenges in organizing the CHIC and SBS Interactive Tracks from 2013 to 2016 in the form of a list of important properties of any future IIR evaluation campaigns.